Kinect-Sign: Teaching Sign Language to "Listeners" through a Game

The sign language is widely used by deaf people around the globe. As the spoken languages, several sign languages do exist. The way sign language is learned by deaf people may have some details to be improved, but one can state that the existing learning mechanisms are effective when we talk about a deaf child, for example. The problem arises for the non-deaf persons that communicate with the deaf persons – the so-called listeners. If, for example, one couple has a new child that turns to be deaf, these two persons find a challenge to learn the sign language. In one hand, they cannot stop their working life, especially because of this sad news turns to be more costly, on the other hand, the existing mechanisms target the deaf-persons and are not prepared for the listeners. This paper proposes a new playful approach to help these listeners to learn the sign language. The proposal is a serious game composed of two modes: School-mode and Competition-mode. The first provides a school-like environment where the user learns the letter-signs and the second provides the user an environment used towards testing the learned skills. Behind the scenes, the proposal is based on two phases: 1 – the creation of a gestures library, relying on the Kinect depth camera; and 2 – the realtime recognition of gestures, by comparing what the depth camera information to the existing gestures previously stored in the library. A prototype system was developed – the Kinect-Sign – and tested in a Portuguese Sign-Language school resulting in a joyful acceptance of the approach. © 2014 The Authors. Published by Elsevier Ltd. Selection and peer-review under responsibility of ISEL – Instituto Superior de Engenharia de Lisboa.